In the emerging market of 3D TV, 3D video, and 3D cinema, many applications require depth information, but their demands concerning the density, accuracy, and reliability of the depth maps differ widely. Therefore, it is beneficial to supplement a disparity map with a confidence map that reflects the reliability of the individual disparity estimates.
One application are VFX (visual effects) using the depth information to model the final scene by combining CGI generated and camera generated movie components. This requires reliable depth information, which preferably is prepared and provided together with the movie. Otherwise a cost-intensive and hand crafted process has to be utilized to generate depth information.
For the computation of depth information from a set of two or more images a matching process is applied to find point correspondences between input images. The displacement between two corresponding points is referred to as disparity. The 3D structure of a scene can be reconstructed from these disparities through triangulation if the camera parameters are known.
A correct and safe exploitation of depth information in video productions depends directly on the quality of disparity estimation generated for stereoscopic or multi-view video sequences. The quality of the calculated depth maps, however, is not at all homogeneously distributed and includes defects whenever the underlying disparity estimation model has been injured. Rapid scene object movements, foreground background occlusions, and missing or periodically structured textures visible in the scenes are some of the well-known origins of flaws in disparity estimation results causing unavoidable quality deficits. Thus, additional information is required to support the expedient application of the gained disparity information.
The mean to indicate possible quality risks is linked to every disparity value and is provided by the disparity confidence values. High confidence values associated with a disparity indicates a safe usage while a low confidence value means the opposite. Selecting depth information from a depth map by making no restrictions to the quality risk in choosing a low threshold for the confident value will result in a dense perhaps complete depth map but will contain many errors. If, in the opposite way, the threshold for the confidence request is increased and only the disparities associated with high confidence are selected the resulting depth map will become sparsely populated but, therefore, more reliable.
The performances of the confidence measures have not been broadly analyzed so far. One reason may be the assumption that a change in the confidence value does not have any effect on the disparity information and that the primary goal must be to improve quality of the disparities. At the other side it is evident that disparity estimation will always be erroneous and that information is urgently needed indicating the locations where this comes true. It is also worth to consider the aspect that the additional confidence information can be used in further post-processing steps for the improvement of the depth maps by adequate modifications.
A state of the art confidence calculation, which combines consistency, visibility, and matching errors to model the reliability state, is described in EP 2 511 875.